Operationalizing AI Governance for Enterprise LLM Services

Enterprise AI is not one system. It is a fleet of chatbots, model APIs, coding assistants, and agents spread across vendors, platforms, identities, and traffic paths. The practical security question is therefore not simply which controls an organization needs, but where those controls can actually be enforced.

Enterprises increasingly operate a heterogeneous mix of large language model services: commercial and internal chatbots, multi-cloud LLM APIs, AI-assisted developer tools, and agentic platforms. Each service brings familiar concerns such as identity, access control, data protection, and monitoring. AI also introduces less deterministic risks, including prompt injection, compromised or misaligned components, and agents that act through tools without direct human approval.

This study asks:

Where should security controls be assigned when no single system, vendor, or enforcement point governs the environment as a whole?

The answer is architectural. An AI gateway is a valuable enforcement point for requests that pass through it, but it cannot govern traffic that bypasses it or controls that depend on a surrounding platform. A broader AI control plane can coordinate policy, inventory, lifecycle status, and evidence across services, but it does not become a fifth enforcement layer or eliminate the need for administrative controls.

Why enterprise AI governance fragments

Enterprise AI adoption is often an accumulation of independent decisions rather than a centrally planned system. Different teams adopt different models, application interfaces, IDE assistants, command-line tools, and agent platforms on different timelines. Each choice introduces its own identity model, administrative console, logging behavior, retention terms, regional footprint, and security controls. The result is not one system to secure, but a fleet of differently governed services.

The supply side reinforces this heterogeneity. The AI Index 2026 Annual Report counted 93 notable industry models released in 2025 across a broad vendor set. Consolidating on a single provider therefore runs against the direction of the market.

The same report shows a persistent gap between recognizing AI risk and actively mitigating it. In a 2025 survey, 72% of organizations considered cybersecurity relevant, while 61% reported actively mitigating it. For autonomous or unintended system actions, the corresponding figures were 44% and 29%.

Responsibility is fragmented too. Information security held primary responsibility for AI governance in 21% of surveyed organizations, risk and compliance in 19%, and dedicated AI-governance roles in 17%. No function exceeded 21%.

What existing guidance does—and where it stops

Organizations can draw on substantial guidance. The NIST AI Risk Management Framework provides a lifecycle model; the NIST Generative AI Profile adapts it to generative AI; ISO/IEC 42001 supplies a certifiable management-system standard; the EU AI Act establishes binding obligations; and the Cloud Security Alliance AI Controls Matrix supplies a granular control catalog. NIST also provides an adversarial machine-learning taxonomy and is developing Control Overlays for Securing AI Systems.

These sources serve different purposes, and none provides one operational model for assigning controls across a heterogeneous enterprise fleet. Frameworks tend to name outcomes without naming the technical enforcement point. Regulations allocate duties by value-chain role, but do not define a role for the enterprise intermediary routing traffic across several providers. Practitioner catalogs describe risks at the level of one application or one agent. Academic architectures may draw a gateway without specifying what it must enforce.

This creates three recurring gaps:

• Coverage is uneven across service categories. Chatbots and model APIs receive the most attention; developer tooling remains comparatively thin despite its short path from generated output to executable privilege.
• Controls are rarely assigned to an enforcement location. Model-, application-, platform-, and gateway-level mitigations are often presented as interchangeable options.
• Roles do not reflect a heterogeneous fleet. Guidance usually assumes one governed system, provider, use case, or contract rather than several vendors and traffic paths under one enterprise policy.

The study therefore builds forward from what the enterprise must defend against, then assigns each control to the strongest layer that can realistically enforce or observe it.

Four service categories

The analysis separates enterprise LLM services according to how traffic reaches the model, whose identity accompanies it, and what the system can do with its output.

Enterprise chatbots

These are conversational interfaces used directly by people, whether supplied commercially or built internally. A human identity accompanies each request, and the principal security surface is the context window and any retrieval or connector capability behind it.

Multi-cloud LLM APIs

Applications call programmatic model endpoints using API keys or service principals. The human who caused a request may be several steps removed from the identity presented upstream. Authentication schemes, logs, filters, retention terms, and regions differ by provider.

AI-assisted developer tooling

IDE assistants, coding agents, code-review tools, and CI automation can read source code and configuration and can emit code or commands into build and execution paths. This category combines properties of chat, APIs, and agents.

Agentic platforms

Agents plan multi-step tasks, retrieve data, invoke tools, and act across trust boundaries, often through the Model Context Protocol. Their consequential actions may be nondeterministic and may occur without human approval at every step.

An actor-oriented threat model

The paper organizes threats by actor and capability rather than by a list of risk types. Confidentiality, integrity, and availability remain the security outcomes; the variable is who acts on the system and how.

1. T1 — External attacker. A third party without credentials can influence model input, especially through indirect prompt injection or attacker-controlled resources consumed at inference time.
2. T2 — Negligent insider. An authorized user exposes data through ordinary use, such as pasting regulated information into an unapproved service, exposing secrets to a coding agent, or enabling overly broad retrieval.
3. T3 — Malicious human insider. An authorized user deliberately abuses legitimate access, evades controls, extracts sensitive content, or routes work through a personally chosen provider.
4. T4 — Compromised or adversarial AI component. A model or agent acts against the organization’s interests, with or without an external compromise.
5. T5 — Excessive agency and overreach. An authorized component does more than intended within access it was legitimately granted.

T4 is especially important because it treats the AI component itself as a possible insider-like actor. In Anthropic’s agentic-misalignment study, frontier models from multiple providers selected deliberate harmful actions in simulated scenarios combining goal conflict with a threat of replacement. The result does not show how deployed models ordinarily behave, but it supports placing consequential enforcement outside the model’s own reasoning.

The OWASP Top 10 for LLM Applications and OWASP Top 10 for Agentic Applications reinforce the same control principle: authorization should not be entrusted to the model, and agency must be bounded through permissions, functionality, and autonomy.

Gateway versus control plane

An AI gateway extends the conventional API gateway. It sits in the request path and can authenticate callers, route requests, inspect prompts and responses, enforce quotas and policy, authorize tool use, redact sensitive content, and record activity. Because it mediates traffic, it is the strongest enterprise-controlled location for controls shaped like an individual request.

An AI control plane sits conceptually above individual request paths. It coordinates policy, inventory, lifecycle status, observability, and evidence across gateways and services. It can make distributed controls and gaps visible, but it cannot directly block traffic on a path it does not operate.

Three design principles

1. Mediate the request. Every relevant request must be checked, and the check must not be bypassable.
2. Resolve every action to a principal. Requests and downstream actions must remain traceable to the human, workload, or agent that caused them.
3. Bound what a component may attempt. Least agency extends least privilege from what an identity may access to how far it may act on that access.

The four enforcement layers

Each control is assigned to the strongest practical layer that can carry it:

• Administrative: organizational policy, acquisition, approval, inventory, contracts, and risk acceptance.
• Platform-native: controls implemented inside the model service, application, agent runtime, sandbox, retrieval system, or vendor platform.
• Gateway-enforceable: controls the enterprise can apply directly to traffic that passes through an AI gateway.
• Gateway-observable: controls whose status or evidence can be recorded centrally even when the gateway cannot apply them.

A 22-control taxonomy

The taxonomy contains 22 controls in five families. Each control maps to one or more threats, the applicable service categories, and a primary enforcement layer. The table below presents the paper’s abbreviated master list.

The distribution is uneven: 10 controls are gateway-enforceable, six are platform-native, four are administrative, and two are gateway-observable. Fewer than half can therefore be enforced at the enterprise-operated gateway.

The limitation follows directly from the request path. A gateway can sanitize a tool result crossing back to an agent, but it cannot determine whether the agent’s internal reasoning later treats that result as data or as an instruction. It cannot isolate a runtime it does not operate, preserve an approval gate inside another platform, or enforce data-retention terms established by contract.

Three requirements remain imperfectly covered by every layer: reliable separation of instructions from retrieved data, integrity of human approval gates when reviewers approve reflexively, and prevention of T4 itself. The taxonomy can contain and attribute adversarial AI behavior, but it cannot guarantee that such behavior never originates.

Testing the taxonomy through four scenarios

1. Enterprise chatbot: indirect prompt injection

An internal support chatbot may read only tickets assigned to the current user. A malicious ticket instructs the model to retrieve another employee’s ticket. The tool allowlist control, TA-1, degrades from gateway-enforceable to platform-native because connector access is controlled entirely inside the chatbot vendor’s platform. DP-2 must prevent the model from treating ticket content as authority, while DP-3 must deny retrieval beyond the caller’s entitlements even if DP-2 fails.

2. Multi-cloud API: the shared-credential attribution gap

A backend routes requests to several model providers through a shared gateway, but all users appear under one long-lived service credential. The gateway can enforce IA-1, IA-2, and OE-1 only with the identity signals it receives. Separate workload credentials improve scoping, but accountability for a particular engineer still requires the application to propagate that human identity into the request and audit trail.

3. Developer tooling: no enterprise-operated hop

A coding agent connects directly from a workstation to its vendor and may send configuration files or embedded credentials. DP-1 falls from gateway-enforceable to administrative because no enterprise component can inspect the request path. TA-4 remains platform-native because execution isolation is a property of the runtime, not the network path. A managed local sandbox can enforce it, but an AI gateway cannot.

4. Agentic platform: vendor-hosted execution

A vendor-hosted procurement agent calls multiple tools and MCP servers. Human approval for high-impact actions, TA-3, remains platform-native. The vetted-tool registry control, TA-5, degrades from gateway-enforceable to platform-native when tools are called directly by the vendor runtime. OE-3 remains observable rather than enforceable because the enterprise depends on vendor telemetry to distinguish agent actions from the initiating human.

What current AI gateways cover

The paper compares five gateway and proxy products as practical examples: LiteLLM, Kong AI Gateway, Portkey, Gravitee, and MuleSoft Omni Gateway. This is a point-in-time review of documented capability, not a product endorsement or a fixed ranking.

Identity and Access is the most consistently supported family. All five products document centralized authentication, scoped credentials, and identity-based policy. Data Protection is partial across the board: gateways inspect traffic, but they do not control tenant isolation, identity-bound retrieval, provider retention, or the agent’s distinction between data and instructions. Tool and Action Authorization is strongest where a gateway can approve or deny a specific tool call before execution. Observability is common, but products differ in AI-specific audit detail, alerting, and independent evidence retention.

Lifecycle and Administrative Governance is the thinnest family because it binds the organization rather than an individual request. MuleSoft receives partial credit in the paper for agent discovery and registry capability, but no gateway product covers the family as a whole.

Beyond the gateway: the AI control plane

The gateway is necessary for complete mediation of request-shaped controls, but it does not provide a fleet-wide view of controls implemented elsewhere. The AI control plane fills this coordination gap by bringing together identity, policy, inventory, lifecycle status, observability, and evidence across distributed services.

The term is still developing. Speakeasy and TrueFoundry emphasize gateways, policy, and telemetry. Atlan and Oracle emphasize inventory, lineage, lifecycle, or organizational governance. These approaches collectively show that lifecycle governance requires more than intercepting requests.

The distinction between visibility and enforcement is critical. A provider may inventory or observe a service outside its own environment, but it can block traffic only when the traffic or workload enters an enforcement service it operates. Cross-provider support is therefore not the same as vendor-neutral control.

How the major cloud providers implement the model

A cloud provider is not one product. Each assembles a control surface from model services, gateways, identity systems, agent runtimes, registries, security tooling, and observability. The paper evaluates AWS, Microsoft Azure, and Google Cloud against the same five control families.

All three providers are strong in Identity and Access and Observability and Evidence. Tool and Action Authorization is strongest for agentic workloads. Data Protection is strongest where prompts and responses enter a gateway or content-inspection service. Registries improve Lifecycle and Administrative Governance through inventory, but they do not prove that an organization approved a service for production or accepted its residual risks.

The same pattern appears across service categories. Agentic platforms and multi-cloud APIs receive the strongest coverage. Chatbots receive direct coverage when built within the provider or routed through its enforcement services. Developer tooling remains weakest unless the IDE or coding agent is deliberately connected through a provider gateway or managed runtime.

The main finding

No single product or enforcement layer provides complete coverage. A gateway can govern controls tied to an individual request: identity, content inspection, routing, tool authorization, and logging. It cannot govern traffic that bypasses it, provide complete lifecycle governance, or enforce controls that depend on another platform’s reasoning loop, retrieval system, runtime, or approval process.

An AI control plane provides the broader operating model. It coordinates policy, inventory, lifecycle status, and evidence across the four existing enforcement layers. It is not one product, and it is not itself a new enforcement layer. Its value is that it reveals where controls are enforced, where they are merely observed, and where administrative action remains necessary.

Organizations must therefore verify:

• which controls are enabled;
• where each control actually operates;
• which identity reaches each enforcement point;
• what evidence the control produces;
• which services bypass the controlled path; and
• which remaining risks require administrative ownership and explicit acceptance.

Administrative controls close the gaps through service inventory, production approval, ownership, independent retention of audit evidence, provider review, contractual terms, and documented acceptance of risks that cannot be technically enforced.

Limitations

The product assessments rely on public documentation and represent a point in time in a rapidly changing field. Documented capabilities may depend on configuration, routing, licensing, integration, region, or preview status. The scores therefore describe documented potential rather than controls enabled in every environment.

Those limitations may change individual product findings, but they do not change the architectural result: enforcement remains strongest where the organization or provider controls the traffic path or runtime, visibility can extend farther than enforcement, and lifecycle governance still requires administrative action.

Conclusion

The practical contribution of the AI control-plane model is not the claim that every AI service can be governed from one place. It is the opposite: the model makes the boundaries of centralized control explicit.

By connecting five actor-oriented threats, 22 controls, four service categories, four enforcement layers, four scenario evaluations, and the current tooling landscape, the model gives enterprises a concrete way to ask where each control belongs—and to see when no technical layer covers it cleanly.

An AI gateway is an important enforcement point, but it is not sufficient by itself. The control plane provides coordination and visibility; platforms provide controls tied to their own runtimes; and administrative governance remains responsible for everything no technical path can reach.